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Cognitive  Technology  Extends  the  Work  Environment 
and  Accelerates  Learning  in  Complex  Jobs 


INTRODUCTION 

Scientists  at  the  Air  Force  Human  Resources  Directorate  of  the  Armstrong 
Laboratory  recently  completed  a  long-term  program  of  research  and  development  (10+ 
years)  investigating  issues  of  technical  competence  in  modem  Air  Force  workcenters.  We 
directed  our  investigation  at  examining  human  problem  solvers  in  real-world,  machine¬ 
laden  work  environments  —  our  goal  being  to  improve  our  understanding  of  the  man- 
machine  interaction  that  results  in  technical  expertise.  The  premise  is  that  with  greater 
understanding,  we  can  build  better  training  and  make  personnel  resource-related  decisions 
that  are  more  responsive  to  high-tech  performance  demands.  Better  training  has  the 
potential  to  accelerate  the  rate  of  complex  skill  acquisition,  and  further,  if  training  focuses 
on  the  components  of  skill  that  are  common  to  a  range  of  complex  tasks,  it  becomes 
possible  to  equip  technicians  with  generalizable  skills  —  i.e.,  a  technical  flexibility  that 
greatly  increases  the  value  of  human  capital  in  an  era  of  rapid  technological  change  and 
diminishing  resources. 


Background 

Our  research  strategy  unfolded  in  four  stages  of  investigation,  as  follows;  (a) 
during  a  pilot  testing  phase,  we  observed  and  interviewed  FI 5  avionics 
technicians/troubleshooters  at  varying  levels  of  proficiency  and  selected  a  separate  sample 
of  airmen  to  participate  in  a  proof  of  concept  experiment;  (b)  based  on  preliminary 
findings,  we  developed,  tested,  refined  and  applied  a  formal  cognitive  task  analysis  (CTA) 
methodology  to  use  in  eliciting  from  experts  the  components  of  skill  needed  for  this 
domain  (Gott,  1987;  Gott,  Bennett,  &  Gillet,  1986;  Hall,  Gott,  &  Pokomy,  in  press; 
Means  &  Gott,  1988);  (c)  we  codified  the  output  of  the  task  analysis  and  then  used  it  as 
input  to  the  development  of  an  intelligent  tutoring  system  for  avionics  troubleshooting; 
and  (d)  we  evaluated  the  tutor  in  a  controlled  experiment  at  three  operational  FI 5  flying 
wings  (Gott  1989;  Gott,  Pokorny,  Kane,  Alley,  &  Dibble,  in  preparation), 

The  results  of  the  cognitive  task  analysis  studies,  which  involved  hundreds  of 
airmen,  revealed  patterns  in  realistic  problem  solving  performance  across  a  range  of 
troubleshooting  tasks  and  human  proficiency  levels.  With  these  data  we  were  able  to 
formulate  a  model  of  technical  performance  that  has  guided  our  intelligent  tutor 
development.  An  abstracted  representation  of  the  model  is  shown  in  Figure  1 .  The  model 
highlights  the  important  interplay  among  (a)  strategy,  (b)  tactics  (procedures)  and  (c) 
conceptual  (system)  knowledge.  The  Strategic  Knowledge  component  sits  on  top  of  and 
controls  the  two  remaining  interactive  components  —  Procedures  (tactics  or  operations) 
and  System  Knowledge.  This  configuration  as  a  model  of  technical  performance  posits 
that  a  top-level  plan  or  strategy  deploys  pieces  of  system  knowledge  and  procedural 


subroutines  as  needed  and  as  driven  by  strategic  decision  factors  such  as  time,  effort, 
payoff,  and  resource  efficiency.  Troubleshooting  is  thereby  represented  as  multilevel, 
complex  decision  making,  which  involves  choices  among  various  top-level  and 
intermediate-level  strategies,  tactics,  and  system  views. 


Figure  1.  Cognitive  Skills  Architecture 


Method 


Tutor  Development 

Given  the  targeted  models  of  expertise  revealed  by  the  cognitive  task  analysis,  i.e., 
electronic  troubleshooting  as  multi-component,  complex  decision  making,  we  knew  that  to 
be  effective  instructionally,  the  learning  environment  had  to  be  robust.  (In  the  real  world, 
expertise  of  this  type  takes  8  to  10  years  to  develop.)  We  adopted  the  following 
multifaceted  principle  of  learning  as  the  foundation  for  instructional  design:  in  complex 
diagnostic  tasks,  mental  models  (system  knowledge)  as  well  as  procedural  and  strategic 
knowledge  are  constructed  as  students  interact  with  the  full  context  of  the  work 
environment,  practicing  shop  procedures  and  fault  isolation  operations  in  response  to 
realistic  troubleshooting  scenarios.  Trainees  receive  support  from  coaching,  which  they 
access  as  needed.  To  culminate  the  process,  learners  reflect  upon  their  solutions 
considering  their  strengths,  diagnosing  their  weaknesses,  and  contemplating  model 
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solutions  of  Masters.  In  short,  this  principle  calls  for  a  situated,  supported  learning 
environment,  which  we  have  termed  coached  apprenticeship. 

Pedagogically,  the  tutor  (called  Sherlock)  functions  as  a  coached,  practice 
environment  where  students  “learn  by  doing”  electronic  troubleshooting.  They  encounter 
high  difficulty  fault  isolation  tasks  which  they  pursue  in  a  computer  learning  environment 
that  is  an  extension  of  their  real  world  work  environment  (situated  learning.)  In  the  real 
world,  technicians  in  this  domain  repair  and  maintain  electronic  subsystems  and  selected 
test  equipment  for  the  FI 5  aircraft.  When  a  line  replaceable  unit  (LRU  or  black  box)  is 
removed  from  the  aircraft  on  the  flightline  because  of  a  suspected  malfunction,  it  is  sent  to 
technicians  in  the  repair  shops.  Upon  arrival  it  is  attached  via  connecting  cables  (or  similar 
apparatus,  referred  to  as  the  Test  Package)  to  a  large  piece  of  test  equipment  known  as  a 
Test  Station.  The  LRU  is  then  referred  to  as  a  Unit  Under  Test,  or  UUT.  Figure  2a 
shows  a  top  level  diagrammatic  view  of  the  complete  equipment  system. 


TEST  STATION 


Figure  2a.  Top-Level  Mental  Model  of  Avionics  Equipment  System 

The  test  station  has  a  wide  range  of  functions  in  accomplishing  its  job  of  (a)  simulating 
the  electronic  signals  (from  a  stimulus  drawer)  that  the  unit  would  receive  if  it  were  in  the 
airplane  and  (b)  measuring  the  signals  the  unit  produces  in  response  (via  a  measurement  drawer 
in  the  station).  Hence  the  station  performs  a  tremendous  array  of  signal  generation,  signal 
routing,  and  signal  measurement  functions.  Further,  troubleshooting  is  complicated  by  the  fact 
that  very  little  of  what  the  test  station  does  is  visible  to  the  technician.  (A  digitized  picture  of 
the  station  is  shown  in  Figure  2b.)  The  opaqueness  and  complexity  of  the  test  station  not  only 
increase  device  knowledge  demands  on  the  performer  but  also  heighten  associated  procedural 
and  strategic  knowledge  requirements  as  well.  To  address  those  performance  demands,  the 
tutor  includes  a  simulation  of  the  actual  equipment  system  and  all  the  functionalities  available  to 


Figure  2b.  Actual  Avionics  Test  Station 
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Figure  3.  Sherlock  Interface 

In  sum,  the  following  critical  features  of  learning  environments  for  complex  skill 
acquisition/generality  provide  the  cornerstone  for  Sherlock: 


-  A  pedagogical  approach  of  learning  by  doing  and  learning  through  reflection  in  a 
situated,  supported  learning  environment  is  indicated. 

-  Situated  learning  supplies  the  needed  context  for  learners  to  execute  tasks  in  an 
environment  that  reveals  the  use  of  their  knowledge. 


-  Supported  learning  enables  a  master-apprentice  instructional  relationship  to  form, 
whereby  the  master  (or  coach)  can  model,  assist,  and  review  problem  solving  performances  as 
well  as  sequence  learning  activities  that  promote  successive  approximations  of  mature  practice. 
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-  Detailed  cognitive  models  and  authentic  problem  solving  scenarios  are  essential  as 
inputs  to  the  instruction  of  complex  problem  solving  skills. 

—  Cognitive  models  make  the  targeted  expertise  explicit,  precise,  and  complete 
(e.g.,  tacit  strategic  knowledge  is  revealed). 

—  Authentic  scenarios  give  the  instruction  validity  and  vigor,  with  their  realism, 
and  they  promote  the  culture  of  expert  practice  in  the  work  environment. 

-  Given  the  multi-component  nature  of  troubleshooting,  all  components  should  be 
treated  in  some  integrated  form  instructionally.  Further,  a  logical  progression  of  models  of 
proficiency  is  needed  to  inform  the  sequencing  of  instructional  events. 

“  Historically,  troubleshooting  training  has  focused  on  a  single  component 
(typically,  the  observable  procedures);  however,  knowing  the  steps  of  procedures  is  a  primitive, 
early  approximation  of  expertise.  Naked,  brittle  actions  have  very  limited  utility  (Rouse,  1982). 

“  Similarly,  device  knowledge  has  been  typically  taught  with  a  focus  on  formal 
theoretical  principles,  without  giving  students  direct  manipulation  experiences  with  the 
phenomenon.  Training  that  has  focused  on  formal  theoretical  principles  without  attention  to 
their  application  has  repeatedly  failed.  Direct  manipulation  experiences  with  devices  promote 
qualitative  causal  reasoning  skills  (Morris  &  Rouse,  1985). 

—  Whaf  s  needed  for  complex  problem  solving  is  instruction  where,  at  the 
minimum,  device  topology  and  procedural  knowledge  are  coordinated  into  robust  conceptual 
device  knowledge,  fi'om  which  procedures  may  be  inferred  if  necessary. 

—  In  addition,  for  the  ill-structured  problems  of  the  real  world,  the  explicit  plans 
and  goals  of  strategic  knowledge  need  to  be  coordinated  with  the  two  other  components  to 
provide  the  glue  that  integrates  the  steps  of  complex  performances. 

To  summarize,  Sherlock  followed  the  pedagogical  tenets  of  (a)  learning  by  doing  in  a 
situated,  supported  learning  environment;  (b)  using  detailed  cognitive  models  as  the  primary 
input  to  instruction;  and  (c)  coordinating  the  multiple  components  of  troubleshooting  expertise 
instructionally. 

Tutor  Evaluation:  Subjects  &  Instruments 

Fifly-four  apprentice,  journeyman,  and  master  avionics  technicians  participated  in  the 
evaluation  study,  which  was  conducted  as  a  controlled  experiment  at  three  geographically 
separated  sites  -  Langley  AFB  VA  Eglin  AFB  FL,  and  Nellis  AFB  NV.  The  apprentice  and 
journeyman  technicians  had  an  average  of  33  months  experience  on  the  job;  the  Masters 
averaged  124  months  (10  years,  4  months)  experience.  Two  types  of  instruments  were  used  to 
assess  learning  and  provide  measures  for  pre  to  post  intervention  comparisons.  To  access  and 
measure  the  covert  processes  and  structures  that  comprise  troubleshooting,  we  developed 
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Verbal  Troubleshooting  Tests  (VTTs)  as  the  principal  learning  assessments  to  evaluate  tutor 
effectiveness.  A  VTT  can  be  conceived  as  a  structured  thinking  aloud  protocol  form  of 
assessment.  It  allows  a  close  approximation  of  hands-on  troubleshooting  performance,  without 
the  costly  and  ineflScient  utilization  of  actual  equipment.  The  second  instrument  is  a 
Noninteractive  Troubleshooting  (NIT)  Test,  which  is  a  paper-and-pencil  instrument  designed 
to  complement  the  VTT.  The  NTT’s  paper  and  pencil  format  eliminates  the  requirement  for 
verbal  responses,  limits  the  involvement  as  well  as  any  biases  of  the  examiner  in  a  VTT,  and 
offsets  the  potential  biasing  influence  of  the  low  number  of  VTT  problems  (two  pretest  and 
three  posttest.) 


Results 

Experimental  and  Control  Group  Comparisons  (Post  Intervention) 

As  predicted,  VTT  and  NIT  scores  revealed  large  and  statistically  significant 
differences  in  favor  of  the  experimental  group  over  the  controls  (Table  1):  VTT  Posttest  #3  {t 
[39]  =  -4.04,  p  =  .000);  VTT  Posttest  #4  {t  [39]  =  -3.72,/?  =  .001);  NIT  Posttest  {t  [39]  = 
-2.77,  p  =  .009).  The  pre-  to  posttest  differences  for  both  tests  are  illustrated  in  Figure  4. 
Comparisons  on  both  tests  for  all  three  groups  (Master  Group  included)  are  illustrated  in 
Figure  5. 

Also,  as  predicted,  the  VTT  and  NIT  posttest  scores  on  a  piece  of  novel  equipment 
(called  Frankenstation)  revealed  statistically  significant  differences  in  favor  of  the  experimental 
group  over  the  controls  (Table  2):  VTT  Posttest  {t  [36]  =  -2.93,  p  =  .006);  NIT  posttest 
(t  [36]  =  -2.34,  p  =  .025).  The  differences  are  illustrated  in  Figure  6. 

Table  1.  Posttest  Measures  of  Troubleshooting  Proficiency 

(Sherlock  2) 


Sherlock  2  Posttests 


Group 

N 

VTT  3 

VTT  4 

NIT 

Novices 

Control 

M_ 

23 

59 

58 

75 

SD 

37 

37 

14 

Experimental 

M. 

18 

95 

91 

87 

SD 

5 

7 

12 

Masters 

M 

13 

85 

86 

86 

SD 
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11 
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Figure  4.  Pre  to  Posttest  Changes:  Sherlock  Tests 
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Figure  5.  Sherlock  Posttest  Scores  Across  Groups 
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Table  2.  Posttest  Measures  of  Transfer 
(Frankenstation) 


Group 

N 

Frankenstation  Posttests 
VTT  NIT 

Novices 

Control 

M 

21 

55 

72 

SD 

31 

4 

Experimental 

M 

17 

82 

80 

SD 

23 

10 

Masters 

M. 

12 

91 

88 

SD 

22 

12 

FRANKENSTATION  FRANKENSTATION 

Verbal  Troubleshooting  Posttests  Noninteractive  Posttests 


Group  Group  Group  Group  Group 

Figure  6.  Frankenstation  Test  Scores  Across  Groups 
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The  effect  size  for  each  of  the  posttest  measures  is  shown  in  Table  3.  The  range  is 
from  .76  to  1.27  standard  deviations.  As  a  basis  for  comparison,  the  average  effect  size  for 
new  science  and  math  curriculum  in  U.S.  schools  is  reported  to  be  .30  standard  deviation 
(Bloom,  1984). 


Table  3,  Effect  Size  for  Posttest  Measures 


Group 


Measure 

Control 

Experimental 

Effect 

Size 

N 

M 

SD 

N 

M 

SD 

VTT  3  (Sherlock) 

23 

59 

37 

18 

95 

5 

1.27  SDS 

VTT  4  (Sherlock) 

23 

58 

37 

18 

91 

7 

1.17  SDS 

NIT  (Sherlock) 

23 

75 

14 

18 

87 

12 

.87  SDS 

VTT  (Frank'tn) 

21 

55 

31 

17 

82 

23 

.96  SDS 

NIT  (Frank'tn) 

21 

72 

11 

17 

80 

10 

.76  SDS 

Qualitative  Analyses  of  Sherlock  2  VTT  Data 

While  the  significant  differences  in  VTT  and  NTT  posttest  scores  (Experimentals  vs. 
Controls)  provide  solid  evidence  of  the  intervention  effect,  it  only  tells  us  that  the  overall 
troubleshooting  performance  of  the  tutored  airmen  improved  after  the  tutoring  sessions.  The 
particular  ways  in  which  their  performance  improved  --  particularly  vis-a-vis  the  guiding 
troubleshooting  principles  embedded  in  the  tutor  —  provides  a  performance  profile  that  is  much 
more  informative.  Toward  that  end,  we  performed  a  componential  analysis  of  the  two  VTT 
posttest  protocols,  focusing  on  the  following  components  of  troubleshooting  skill  (which 
correspond  to  the  troubleshooting  principles  advocated  by  experts  and  emphasized  in  the 
tutor)  (Gott  et  al,  in  preparation); 

(1)  Safety  of  procedures 

(2)  Accuracy  of  investigating  active  components  in  electronic  test 

(3)  Accuracy  of  measurements  performed 

(4)  Thoroughness  of  component  testing 

(5)  Verification  of  faulty  component  prior  to  swapping 

(6)  Logic  of  sequence  in  which  functional  areas  investigated 

(7)  Accuracy  of  inferences  based  on  test  results 
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We  also  analyzed  six  other  performance  components  of  interest: 

(1)  Total  number  of  troubleshooting  steps 

(2)  Number  of  measurements  made 

(3)  Number  of  ohms  measurements  made 

(4)  Number  of  voltage  measurements  made 

(5)  Number  of  swaps 

(6)  Number  of  components  investigated. 

Results  from  these  analyses  are  shown  in  Figures  7  and  8. 


■  Control  Group  □  Experimental  Group 


measurement  sequence 


Figure  7.  Violations  of  Sherlock  Troubleshooting  Principles 

by  Group 

The  experimental  group  performed  significantly  better  than  controls  on  every 
troubleshooting  principle,  with  the  exception  of  unsafe  actions  (Figure  7).  The  difference  is 
especially  dramatic  on  the  much  lower  frequency  of  swaps  by  experimentals,  which  is 
essentially  a  default  action  executed  by  technicians  when  they  have  no  productive  action  to 
take.  Other  experimental-control  comparisons  (Figure  8)  reveal  (a)  no  difference  in  mean 
number  of  actions  taken  but  (b)  fewer  components  and  circuits  investigated  by  experimentals, 
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and  (c)  significantly  more  voltage  measurements  (more  difficult  measurements  to  execute  and 
interpret)  taken  by  experimental  subjects. 


^Control  Group  □Experimental  Group 


24 


actions  measurements  measurements  actions  investigated  investigated 


Figure  8.  Comparisons  on  Additional  Sherlock  Components 


Discussion 

How  has  Sherlock  2  been  able  to  achieve  this  level  of  effectiveness?  Although  our 
study  did  not  include  conditions  where  instructional  features  were  manipulated  to  gauge  the 
power  of  individual  instructional  attributes,  past  research  in  the  training  of  complex  problem 
solving  skills  lends  support  to  certain  speculative  explanations  regarding  Sherlock's 
effectiveness.  The  support  centers  around  (a)  cognitive  models  as  input  to  instruction;  (b)  the 
sequencing  of  instructional  events;  (c)  situated  learning  in  a  constructivist  instructional 
environment,  and  (d)  the  sociology  of  a  learning  system.  Topics  (c)  and  (d)  fall  within  the 
scope  of  this  paper. 

Situated  Learning  in  a  Constructivist  Environment 

Following  Dewey  and  in  accord  with  the  current  constructivist  movement  (Perkins, 
1991),  the  general  principle  of  learning  by  doing  is  the  touchstone  of  the  Sherlock  design. 
The  tutor  is  an  extension  of  the  trainee's  actual  work  environment.  Authentic  fault  isolation 
problems  are  selected  and  presented  to  students  as  holistic  scenarios  to  solve  as  they  actively 
construct  their  understanding  of  the  equipment  and  of  the  troubleshooting  task.  Working  in 
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Sherlock  is  like  doing  one's  job  in  the  real  world  -  objects  in  the  environment  are  acted  upon  to 
achieve  certain  goals.  There  are,  however,  several  nontrivial  bonuses  in  Sherlock  that  do  not 
exist  in  the  real  world. 

First,  in  the  actual  shop  environment,  trainees  must  learn  about  test  station 
troubleshooting  as  opportunities  present  themselves.  Unfortunately,  the  opportunities  are 
infrequent,  because  the  frequency  of  station  malfunction  is  relatively  low.  However,  being  able 
to  successfully  troubleshoot  the  test  station  is  a  critical  skill  for  this  specialty,  commanding  high 
training  emphasis.  Further,  since  malfunctions  occur  essentially  at  random,  learning  is  driven 
by  whatever  breaks.  Instruction  cannot  be  sequenced  in  the  manner  just  described  where 
movement  through  upwardly  compatible  models  of  understanding  and  performance  can  be 
fostered.  Finally,  it  is  possible  in  Sherlock  to  time-compress  the  routine  activities  that  may  take 
an  inordinate  amount  of  time  in  the  real  world  so  that  valuable  instructional  time  is  devoted  to 
the  challenging  part  of  the  task. 

The  second  bonus  is  that  the  Sherlock  environment  is  forgiving;  mistakes  can  be  made 
without  dire  consequences,  plus,  expert  coaching  is  always  available  as  scaffolding,  when 
needed.  Scaffolding  in  a  learning  environment  supports  trainees  as  they  try  to  make  sense  of 
the  domain  --  with  hints,  explanations,  even  missing  pieces  of  knowledge.  In  Sherlock ,  the 
scaffolding  appears  as  coaching  during  problem  solving,  and  additional  support  is  provided  in 
reflective  followup  (RFU)  activities. 

In  the  Sherlock  RFU,  students  engage  in  four  activities  designed  to  foster  learning 
through  reflection:  (a)  they  view  a  replay  of  the  solution  steps  they  just  executed;  (b)  they  see 
their  solution  juxtaposed  to  an  exemplar  Master  solution;  (c)  they  can  view  a  more  elaborated 
replay  of  a  Master  solution;  and  (d)  they  are  asked  to  diagnose  their  own  solution  trace  using 
the  principles  of  good  troubleshooting.  Their  self-diagnosis  is  then  compared  to  the  computer 
coach's  diagnosis,  for  validation.  The  RFU  culminates  the  problem  solving  session,  and,  we 
believe,  provides  a  considerable  value-added  element  to  the  Sherlock  1  system  where  there  was 
no  post-performance  review  capability. 

By  enhancing  earlier  learning-by-doing  pedagogy  with  a  leaming-by-reflection 
component,  we  believe  the  instructional  impact  has  been  heightened.  As  Collins  and  Brown 
(1988)  and  Collins,  Brown,  &  Newman  (1989)  have  argued,  learning  through  reflection  in  a 
computational  environment  such  as  Sherlock  achieves  the  following  objectives; 

(1)  The  student's  solution  trace  becomes  a  useful  object  of  study,  especially  since  the 
computer  can  represent  the  process  of  the  solution  and  thereby  externalize  decisions  for 
interpretation  from  a  variety  of  perspectives. 

(2)  By  having  access  to  an  exemplar  solution  from  a  Master  (including  the  Master  s 
normally  tacit  reasons  for  each  action),  the  student  can  observe  and  even  discover  expert 
strategies  and  reasoning  that  subsequently  can  improve  the  trainee  s  own  solution. 
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(3)  After  viewing  a  number  of  problem-specific  traces  in  the  RFU,  the  student  can 
derive  abstractions  from  the  patterns  of  actions  and  underlying  reasons. 

(4)  By  treating  each  of  his  traces  as  useful  objects  of  study,  the  student  can  come  to 
view  learning  as  an  "incrementally  staged  process"  that  happens  over  time,  not  all  at  once 
(Collins,  Brown,  &  Newman,  1987).  Further,  the  self-diagnosis  activity  provides  the  student 
with  concrete  benchmarks  of  his  own  progress  along  the  skill  continuum. 

(5)  The  self  diagnosis  that  occurs  can  then  become  internalized  by  the  student  as  a 
form  of  self-correction,  self-monitoring  capability.  These  are  metacognitive  skills  the  trainee 
may  not  have  possessed  before  the  Sherlock  experience. 


Sociology  of  the  Learning  Environment 

Above  all  else,  instruction  must  be  viewed  as  valid  to  trainees.  It  must  serve  their 
needs,  profit  them  directly.  "Drawing  students  into  a  culture  of  expert  practice  in  cognitive 
domains  involves  teaching  them  to  think  like  experts"  (Collins,  Brown,  &  Newman,  1989, 
p.  488).  In  effect,  Sherlock  seeks  to  do  just  that  —  draw  students  into  a  culture  of  expert 
practice,  enable  them  to  reach  the  mature  levels  of  proficiency  they  observe  being  practiced  by 
the  masters  who  are  the  acknowledged  leaders  in  the  shop. 

On  a  daily  basis  during  the  field  trials  we  observed  the  tutored  subjects  make  strides  in 
becoming  a  part  of  the  community  of  expert  practice.  They  shared  with  us  conversations  they 
had  with  their  Team  Leaders  when  malfunction  problems  arose  on  the  actual  equipment  that 
were  covered  in  Sherlock.  They  sometimes  consulted  the  acknowledged  masters  in  the  shop 
(after  a  tutoring  session)  when  they  needed  and  wanted  more  elaboration  about  a  Sherlock 
scenario  than  was  available  in  the  tutor.  There  were  also  occasions  during  tutoring  sessions 
when  trainees  would  want  to  get  an  opinion  from  one  of  the  shop  7-levels  because  they 
thought  the  tutor's  interpretation  or  suggestion  was  too  narrow  or  incomplete. 

For  some  of  the  more  experienced  apprentice  subjects,  they  sometimes  opted  to  use  the 
device  simulation  in  the  tutor  as  a  way  to  discover  the  equipment's  functionality  and  observe 
general  cause  and  effect  relationships.  They  did  this  by  making  additional  measurements  during 
a  Sherlock  scenario  (i.e.,  more  than  needed  to  solve  the  problem)  just  because  they  wanted  to 
verify  or  expand  their  device  knowledge.  They  explained  that  when  they  first  started  working 
on  another  test  station,  e.g.,  the  Electronic  Warfare  Station,  they  would  do  just  that  -  take  a 
lot  of  measurements  as  a  way  of  figuring  out  how  the  system  worked. 

These  observations  provide  one  measure  of  Sherlock's  effectiveness  in  socializing 
apprentices  into  the  expert  culture.  On  the  Tutor  Report  Card,  trainees  also  reported  increased 
confidence  in  performing  the  hardest  tasks  in  their  job.  The  argument  can  be  made  then  that 
Sherlock  was  viewed  as  valid  instruction  by  trainees.  They  were  learning  to  perform  tasks  that 
they  recognized  as  having  value  in  the  shop,  in  their  culture.  Moreover,  the  acquiring  of  skill 
and  knowledge  from  Sherlock  was  enabling  them  to  be  more  conversant  with  shop  Masters 
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about  the  domain.  In  short,  their  status  in  their  culture  was  on  the  rise  and  it  is  reasonable  to 
assume  they  attributed  some  of  that  ascendance  to  Sherlock. 

Summation 

In  considering  the  acquisition  of  complex  skills  on  real  world  tasks,  we  have  posed  two 
major  premises  as  foundation.  First,  skill  acquisition  in  practical  domains  depends  upon 
purposeful  learning  experiences  where  knowledge  connects  with  its  uses  in  the  world. 

(Cognitive  technology  makes  that  nexus  possible.)  Secondly,  cognitive  modeling  makes 
explicit  the  forms  and  utilities  of  knowledge  that  may  otherwise  go  unobserved,  untaught,  and 
therefore  unlearned.  These  premises  have  come  to  the  forefront  of  practical  skills  training  as  a 
result  of  several  significant  educational  and  technological  trends.  Educational  systems  at  all 
levels  appear  to  have  gradually  weakened  the  ties  between  the  knowledge/skills  (that  are  the 
province  of  formal  schooling)  and  their  uses  in  the  world.  A  renewed  interest  in  apprenticeship 
instruction  and  related  empirical  work  in  learning  suggest  that  the  pedagogical  principles  that 
characterize  classic  apprenticeship  methods  can  help  to  remedy  this  discontinuity. 

Further,  as  rapidly  advancing  technologies  increase  the  number  of  mental  (vs.  manual) 
tasks,  critical  elements  of  the  expert’s  performance  are  correspondingly  more  likely  to  become 
unobservable  to  the  apprentice.  Mental  processes  and  features  of  knowledge  often  remain 
tacit,  that  is,  unarticulated  and  therefore  unknowable.  Cognitive  models  of  real-world  task 
performance  make  the  elements  and  processes  of  modern-day  expertise  explicit,  observable  and 
therefore  potentially  leamable. 

In  practical  domains,  the  modeled  expertise  has  revealed  the  multiple  sources  and  levels 
of  knowledge  that  experts  bring  to  bear  on  the  types  of  complex,  ill-structured  problems  that^ 
are  commonly  encountered  in  modern-day  workplaces.  A  major  finding  concerns  the  expert’s 
capability  to  engage  in  adaptive,  opportunistic  reasoning  that  involves  the  coordination  of  three 
major  sources  of  knowledge;  procedural,  device  (or  system),  and  strategic  control  knowledge. 
For  each  type  of  knowledge,  the  expert  has  access  to  elaborate  abstraction  hierarchies  that 
range  from  specific  knowledge  instantiations  to  abstractly  stated  principles. 

A  second  major  conclusion,  which  is  supported  by  a  growing  body  of  empirical 
evidence,  is  that  skill  acquisition  occurs  through  successive  approximations  of  the  targeted 
expertise.  The  progression  is  characterized  by  movement  from  partial  to  more  complete  (and 
thus  complex)  (a)  understandings  of  domain  phenomena,  (b)  procedural  subroutines,  and  (c) 
strategic  control  structures,  including  goals,  plans,  and  decision  factors.  The  apprentice 
achieves  various  levels  of  incomplete  knowledge  and  capability  on  the  way  to  mastery. 
Principles  and  methods  from  classic  apprenticeship  training  are  proving  useful  in  fostenng  this 
progression.  Situated,  supported,  and  carefully  devised  and  sequenced  learning  expenences 
have  been  shown  to  foster  development  in  veridical  learning  environments  that  cognitive 
technology  has  borne. 

Concordant  with  these  premises,  I  report  results  from  the  recent  controlled  experiment 
conducted  to  evaluate  a  piece  of  US.  Air  Force  cognitive  technology  -  the  avionics 
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troubleshooting  tutor  called  Sherlock.  Outcomes  of  situated  and  supported  learning  in  this 
tutor  show  dramatic  gains  in  fault  isolation  proficiency,  as  well  as  in  skill  generality  by 
apprentice  technicians.  With  the  Sherlock  (computer  tutor)  system,  authentic  fault  isolation 
scenarios  are  presented  in  a  computer  environment  that  is  an  extension  of  the  airman’s  real 
work  environment.  Learning  is  meaningful  because  the  trainee  works  on  problems  he  sees  the 
acknowledged  experts  in  his  culture  confi-ont  every  day.  He  observes  his  own  incremental 
growth  and  eventual  movement  into  this  same  community  of  expert  practice.  He  even  gains 
confidence  he  can  transfer  what  he  knows  to  novel  equipment  systems.  A  victory  for 
successful  learning  in  the  modem  workplace  using  cognitive  technology  can  be  declared. 

Three  advances  in  cognitive  science  have  been  central  to  the  realization  of  this 
instmctional  system;  (a)  cognitive  performance  models,  which  Newell  and  Simon  (1972) 
said  over  20  years  ago  must  be  given  precedence  in  research  (over  studies  of  learning 
processes)  so  that  a  more  complete  and  successful  theory  of  learning  and  development  can 
ultimately  emerge;  (b)  a  cognitive  task  analysis  methodology  robust  enough  to  codify  the 
expert  performance  models,  as  well  as  the  successive  approximations  of  expertise  encountered 
along  the  way;  and  (c)  the  “mental  experiments”  that  experts  run  to  simulate  explanatory 
models  (“in  the  mind’s  eye”)  during  diagnostic  and  procedural  tasks.  The  confluence  of  these 
and  other  advances  have  taken  us  a  step  forward  in  fostering  learning  in  the  real  world  via 
engaging  applications  of  cognitive  technology. 
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